Method and apparatus for evaluating visitors to a web server

ABSTRACT

Different web pages on a web server are associated with different qualification profiles, each of which is assigned a value by the web-site proprietor. Traffic data hits at the web-site are analyzed to determine which web pages the visitor viewed on the web server. Each qualifying visitor is thereafter associated with a qualification profile and a corresponding value. In another aspect of the invention, visitors arriving as a result of an advertisement on a remote web-site are tracked. The web-site proprietor is consequently able to determine a return on advertising investment based on the value of visitors brought to the site by the tracked advertisement.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.09/240,208, filed Jan. 29, 1999, now U.S. Pat. No. 6,925,442.

BACKGROUND OF THE INVENTION

The present invention relates generally to web-server traffic dataanalysis and more particularly to a system and method for determiningthe value of visitors to a web site.

The worldwide web (hereinafter “web”) is rapidly becoming one of themost important publishing mediums today. The reason is simple: webservers interconnected via the Internet provide access to a potentiallyworldwide audience with a minimal investment in time and resources inbuilding a web site. The web server makes available for retrieval andposting a wide range of media in a variety of formats, including audio,video and traditional text and graphics. And the ease of creating a website makes reaching this worldwide audience a reality for all types ofvisitors, from corporations, to startup companies, to organizations andindividuals.

This recent growth of the Internet over the past few years has openednew markets for business. Individuals use the Internet for everythingfrom buying new cars to ordering a pizza to hiring a plumber. The easewith which people can use the Internet for such activities has spurredbusinesses to offer the services and products people desire on theInternet.

Unlike other forms of media, a web site is interactive, and the webserver can passively gather access information about each visitor byobserving and logging the traffic data exchanged between the web serverand the visitor. Important facts about the visitors can be determineddirectly or inferentially by analyzing the traffic data and the contextof the “hit.” Moreover, traffic data collected over a period of time canyield statistical information, such as the number of visitors visitingthe site each day, what countries, states or cities the visitors connectfrom, and the most active day or hour of the week. Such statisticalinformation is useful in tailoring marketing or managerial strategies tobetter match the apparent needs of the audience. Each hit is alsoencoded with the date and time of the access.

Visitors to a web site are not of uniform interest to the site operator.For example, a relatively low quality visitor might be one that merelyreads the home page of the site and moves on. A higher quality visitormight be one that locates, e.g., a product description page, and an evenhigher quality visitor might be one who visits the price page. Thehighest quality visitor is, of course, one that orders and pays forgoods or services offered by the web site.

Visitors may be induced to visit a web site via advertisements placed onremote web sites. One common way to advertise on the Internet is withbanner ads. A banner ad shows a picture or statement of a business'sproducts and services and allows a visitor to click on the ad to visitthe web site hosted by the business. The visit may be to obtain moreinformation, or, as is hoped by the business, to effect a purchase of aproduct or service via the web site.

The problem with Internet advertising is measuring its effectiveness.Internet advertising campaigns vary in price, depending on severalfactors such as where the ad is to be placed and the expected viewingpopulation. One advertising campaign might cost a business, for example,$1,000.00 to be on display for one month at one site, whereas the samead might cost only $500.00 at another site. Further, it has beendifficult to determine exactly how many visitors an advertising campaigngenerated. In the above example, the $500.00 advertising campaign mighthave generated 10,000 visits in the month the ad was up, whereas the$1,000.00 advertising campaign might have generated only 7,000 visits amonth. Finally, the value an advertising campaign generates is afunction of what happens after the visitors visit the site. In the aboveexample, although the $500.00 advertising campaign generated more hits,if none of those visitors made any purchase of the business's productsor services, their collective value would be close to $0.00. On theother hand, although the $1,000.00 advertising campaign generated fewerhits, those visitors might all have purchased products or services,making their collective value at least several hundreds of thousands ofdollars. It is this difficulty in gauging the effectiveness ofadvertising campaigns that makes their use a gamble. Accordingly, thereremains a need for a way to analyze the effectiveness of advertisingcampaigns to determine their relative worth.

It would be desirable to analyze the information gathered by the webserver to determine the quality of visitors to the web site.

It would also be desirable for the site operator to use the determinedvisitor quality to analyze the effectiveness of the operator'sadvertising.

SUMMARY OF THE INVENTION

A business owner indicates what advertising campaigns that are run forthe web site of interest. The business owner also constructs profilesthat define products or services the business owner wants to sell, andassigns each profile a value. The software then analyzes the web site'slog files (which track every exchange of traffic data between the website and other computers over the Internet) and matches visitors withboth an advertising campaign and a profile. Finally, the software sumsthe value for each visitor matched with an advertising campaignaccording to the profiles with which the visitor is matched.

One object of the present invention is to allow an Internet businessowner a way to track the value of visitors who visit the web site.

Another object of the present invention is to allow an Internet businessowner a way to calculate the return on investment for each advertisingcampaign the business owner is currently running.

The foregoing and other objects, features, and advantages of the presentinvention will become more readily apparent from the following detaileddescription of a preferred embodiment that proceeds with reference tothe drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows the communication between a remote visitor, a first website, and a second web site, where the second web site refers thevisitor to the first web site.

FIG. 2 shows a prior art format used in recording a “hit” of trafficdata received by the server of FIG. 1.

FIGS. 3 and 4 illustrate hits memorialized in the log file of the serverin FIG. 1.

FIGS. 5 and 6 show an embodiment for data structures that storequalification profiles and advertising campaigns.

FIG. 7 shows a flow diagram for processing records of traffic data hitsand calculating the return on investment for advertising campaigns in afirst embodiment of the invention.

FIG. 8 shows a flow diagram for assigning a traffic data hit to avisitor session in the first embodiment.

FIG. 9 shows a flow diagram for assigning visitor sessions toqualification profiles based on the visitor's activity in the firstembodiment.

FIG. 10 is a second embodiment that shows a flow diagram for processingtraffic records offline from a data source, such as a log file,evaluating the traffic, and correlating the value to any correspondingad campaigns.

FIG. 11 shows a flow diagram for closing and accumulating visitorsessions in the second embodiment.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

FIG. 1 is a functional block diagram of a prior art system for analyzingtraffic data in a distributed computing environment 9. It is more fullydescribed in “WebTrends Installation and Visitor Guide,” version 2.2,October 1996, and in U.S. patent application Ser. No. 08/801,707, filedon Feb. 14, 1997, the disclosures of which are incorporated herein byreference for all purposes. WebTrends is a trademark of WebTrendsCorporation, Portland, Oreg.

A first server 10 provides web site and related services to remotevisitors. By way of example, the remote visitors can access the server10 from a remote computer system 12 interconnected with the server 10over a network connection 13, such as the Internet or an intranet, adial up (or point-to-point) connection 14 or a direct (dedicated)connection 17. Other types of remote access connections are alsopossible.

Each request by a remote visitor to the server 10—and the replythereto—comprises a “hit” of raw traffic data 11. The format used inrecording each traffic data hit 11 and an example of a traffic data hit11 are described below with reference to FIGS. 2-4. The server 10preferably stores each traffic data hit 11 in a log file 15, although adatabase 16 or other storage structure can be used. The presentinvention can be implemented using any source of traffic data hits.

The present embodiment of the invention is implemented in part via acomputer program run on server 10. It should be appreciated, however,that the invention could be implemented as well by a program operatingon a different computer, such as a work station connected to server 10or a computer connected to server 10 via an intranet or other network.As will shortly be seen, the computer implementing the invention neednot even be connected to the server; it is necessary only that thetraffic data hits generated by the server be accessible by the computerimplementing the invention. For example, the computer implementing theinvention could monitor and process the traffic directly with networkconnection 13.

Five sources of traffic data hits 11 (remote system or visitor 12,dial-up connection 14, log file 15, database 16 and direct connection17) are shown. Other sources are also possible. The traffic data hits 11can originate from any single source or from a combination of thesesources. The program examines each traffic data hit 11 and analyzes theaccess information obtained from the traffic data in a manner laterdescribed in more detail.

In the described embodiment, the server 10 is typically an IntelPentium-based computer system equipped with a processor, memory,input/output interfaces, a network interface, a secondary storage deviceand a user interface, preferably such as a keyboard and display. Theserver 10 typically operates under the control of either the MicrosoftWindows NT or Unix operating systems and executes either MicrosoftInternet Information Server or NetScape Communications Server software.Pentium, Microsoft, Windows, Windows NT, Unix, Netscape and NetscapeCommunications Server are trademarks of their respective owners.However, other server 10 configurations varying in hardware, such asDOS-compatible, Apple Macintosh, Sun Workstation and other platforms, inoperating systems, such as MS-DOS, Unix and others, and in web softwareare also possible. Apple, Macintosh, Sun and MS-DOS are trademarks oftheir respective owners.

A second server 8 also provides web site and related services to remotevisitors such as visitor 12. It is constructed and operates insubstantially the same manner as first server 10 and can be accessed inthe same fashion.

The present invention is preferably implemented as a computer programexecuted by the server 10 and embodied in a storage medium comprisingcomputer-readable code. In the described embodiment, the program iswritten in the C programming language, although other programminglanguages are equally suitable. It operates in a Microsoft Windowsenvironment and can analyze Common Log File, Combined Log File andproprietary log file formats from industry standard web servers, such asthose licensed by NetScape, NCSA, O'Reilly WebSite, Quarterdeck,C-Builder, Microsoft, Oracle, EMWAC, and other Windows 3.x, Windows NT95, Unix and Macintosh Web servers.

FIG. 2 shows a format used in storing a “hit” of raw traffic data 11received by the server of FIG. 1. A raw traffic data hit 11 is not inthe format shown in FIG. 2. Rather, the contents of each field in theformat is determined from the data exchanged between the server 10 andthe source of the traffic data hit 11 and the information pulled fromthe data is stored into a data hit using the format of FIG. 2 prior tobeing stored in the log file 15 (shown in FIG. 1) or processed.

Each traffic data hit 11 is a formatted string of ASCII data. The formatis based on the standard log file format developed by the NationalCenter for Supercomputing Applications (NCSA), the standard loggingformat used by most web servers. The format consists of seven fields asfollows:

Field Name Description Visitor Internet protocol (IP) address or domainname of the Address (30): visitor accessing the site. RFC931 (31):Obsolete field usually left blank, but increasingly used by many webservers to store the host domain name for multi-homed log files. VisitorExchanges the visitor name if required for access to the Authenticationweb site. (32): Date/Time Date and time of the access and the timeoffset from (33): GMT. Request (34): Either GET (a page request) or POST(a form submission) command. Return Code Return status of the requestwhich specifies whether the (35): transfer was successful. Transfer SizeNumber of bytes transferred for the file request, that is, (36): thefile size.

In addition, three optional fields can be employed as follows:

Field Name Description Referring URL URL used to obtain web siteinformation for performing (37): the “hit.” Agent (38): Browser version,including make, model or version number and operating system. Cookie(39): Unique identifier permissively used to identify a particularvisitor.

Other formats of traffic data hits 11 are also possible, includingproprietary formats containing additional fields, such as time totransmit, type of service operation and others. Moreover, modificationsand additions to the formats of raw traffic data hits 11 are constantlyoccurring and the extensions required by the present invention to handlesuch variations of the formats would be known to one skilled in the art.

In operation, remote visitor 12, whose Internet address might bevisitor.sample.org, clicks on a link to request a web page from thesecond web server 8, whose Internet address might be www.portal.com.(Another name for a web page address in Internet parlance is UniversalResource Locator, or URL.) This click generates a traffic data hitconsisting of a reply. In this case, the traffic data hit is a requestfor the second web server 8 to provide to the remote visitor 12 the webpage http://www.portal.com/somepage.htm. Since the remote visitor 12generated the request, the traffic data hit is a “GET” command. Uponreceiving the traffic data hit, the second web server 8 sends a replyback to the remote visitor 12 consisting of an “OK” message and therequested web page. The computer of visitor 12 then displays therequested web page http://www.portal.com/somepage.htm on the remotevisitor's 12 browser.

Somewhere on the provided web page might be an advertisement for thefirst web server, whose Internet address is www.example.com. Thisadvertisement is an advertising campaign run by the first web server 10.If the remote visitor is interested in finding out more about theproducts or services advertised, she might then click on theadvertisement, which includes a link that redirects the visitor to thefirst web server 10. This generates a second request for a web page.This request is directed to the first web server 10. By clicking on theadvertisement, the visitor requests the first web server 10 to providethe web page http://www.example.com/portal_ad.htm via a second trafficdata hit 11 a, shown in FIG. 3. It should be clear that the requestedweb page http://www.example.com/portal_ad.htm in traffic data hit 11 ais on the first web server 10, and not on the second web server 8. Uponreceiving the traffic data hit 11 a requesting web pagehttp://www.example.com/portal_ad.htm, the first web server 10 sends databack to the remote visitor 12 containing an “OK” message and therequested web page. The first web server also writes an entry in its logfile memorializing the request for web pagehttp://www.example.com/portal_ad.htm. This entry 11 a stores severalimportant pieces of information. Entry 11 a stores the remote visitor'sInternet address (visitor.sample.org), the time and date of the request([Jan. 12, 1996:20:37:55 +0000], or Jan. 12, 1996, at 8:37:55 PM,Greenwich Mean Time), the request issued by the remote visitor 12(“GET/portal_ad.htm HTTP/1.0”), and the referring URL(http://www.portal.com/somepage.htm). (Note that, although it is notshown, the second web server 8 can also create its own log file entriessimilar to those of log file 15.)

After being referred to server 10 as described above, remote visitor 12visits several pages on server 10. In FIG. 4, hits are depicted thatrepresent visits by the remote visitor, whose Internet address isvisitor.sample.org, to these pages on server 10. The remote visitor 12first generates a traffic data hit 11 b, requesting the web pagedefault.htm. When the web server 10 receives the traffic data hit 11 b,the web server replies by sending data containing a response message,along with the requested web page default.htm. The web server alsowrites entry 11 b into its log file 15. The log file entry 11 bindicates remote visitor's 12 request for the web page default.htm.

A little while later, the remote visitor 12 requests web pageproducts.htm in traffic data hit 11 c. The web server 10 receives thetraffic data hit 11 c and sends back data containing a response messageand the requested web page products.htm. The web server 10 also writesanother entry 11 c into the log file 15. This log file entry 11 cindicates the remote visitor's request for the web page products.htm.

FIG. 5 shows an embodiment for the qualification profiles. Generallyspeaking, a qualification profile comprises predetermined activity at aweb site. It is used to correlate a pattern of activity with a value.The pattern of activity could indicate a level of interest in a productor service offered at the web site. In FIG. 5, the qualification profiledata structure 300 is shown holding two qualification profiles 305 and310; however, the invention can be implemented with any number ofqualification profiles. The names for the qualification profiles areassigned by the site operator, and may be more meaningful than thearbitrary names Qualification profile 1 and Qualification profile 2 usedhere. For example, they might be the names of different products orclasses of products offered for sale by the operator of the site onserver 10. Qualification profile 305 includes three qualification levels315, 320, and 325; qualification profile 310 includes two qualificationlevels 330 and 335. Each qualification level represents a differentlevel of interest in the product or service represented by thequalification profile. Although the preferred embodiment has threequalification levels—fully qualified, moderately qualified, andminimally qualified—the invention may be implemented with any number,and as shown in FIG. 5, different qualification profiles can havedifferent numbers of qualification levels.

As each qualification level includes the same type of information, onlyqualification level 315 will be discussed in greater detail, althoughFIG. 5 shows the detail for the other qualification levels 320, 325,330, and 335. Qualification level 315 includes two elements: a set ofrequirements 340 (or entry criteria) a visitor must meet to qualify forthe qualification level in that qualification profile, and aquantitative value 345 to assign to each visitor who qualifies for thatqualification level. It should be appreciated that the value associatedwith a qualification level need not be quantitative: it may simply beinherent in the requirements of the qualification level.

The set of requirements 340 a visitor must meet to qualify for thatqualification level in that qualification profile are very flexible.First, in the preferred embodiment, the sets of requirements 340 do nothave to be related in any way. For example, it is not required that theset of requirements for qualification level 315 be a proper subset ofthe set of requirements for qualification level 325.

In the preferred embodiment, the sets of requirements 340 can be tied toweb pages (or URLs) a visitor must visit. Requirements 340 could also betied to the elapsed time the visitor spends at the site or on aparticular page or group of pages, or to information submitted by avisitor in a form located on one of the web pages. Another example ofrequirements 340 includes visiting a content group, which is simply acollection of pages or a class of URLs. In addition, one of requirements340 could be a return visit by a visitor. It should be appreciated thatthe site operator provides such information as the name of thequalification profiles 305 and 310, the chosen qualification levels 315,320, 325, 330, and 335, the sets of requirements 340, and thequalitative value 345 to assign to each visitor who qualifies for thequalification level in that qualification profile.

Not shown in FIG. 5 is the nonqualified visitor value that is assignedto each visitor who does not qualify for any qualification profile.

FIG. 6 shows a data structure for the advertising campaigns. In FIG. 6,the advertising campaign data structure 375 is shown holding threeadvertising campaigns 377, 378, and 379; however, there is norequirement that there be any particular number of advertisingcampaigns. The names for the advertising campaigns are assigned by thesite operator, and may be more meaningful than the arbitrary namesAdvertising Campaign 1, Advertising Campaign 2, and Advertising Campaign3 used here. For example, in the case of tracking banner advertisementsthat refer visitors from other web sites, the name of the referring sitemight be used. As each advertising campaign includes the same type ofinformation, only advertising campaign 377 will be discussed in greaterdetail, although FIG. 6 shows the detail for the other advertisingcampaigns 378 and 379. Advertising campaign 377 includes five elements:an entry page 380 (an entry page is the first hit in a visitor session:in FIG. 1, the web page http://www.example.com/portal_ad.htm is theentry page), a referrer 383 (a referrer is a web page that links avisitor to the advertised web site: in FIG. 1, the web pagehttp://www.portal.com/somepage.htm is the referrer), a start date 386for the advertising campaign, an end date 389 for the advertisingcampaign, and a cost 392 for the advertising campaign during the periodfrom the start date 386 to the end date 389. The entry page 380 and thereferrer 383 help the program associate each visitor with an advertisingcampaign. To make it possible to distinguish between advertisingcampaigns, entry pages 380 and referrers 383 should be unique. Althoughnot required, in the preferred embodiment the entry page 380 and thereferrer 383 are mutually exclusive elements: the operator can assignonly one of the two, and the other remains blank.

FIG. 7 illustrates a first embodiment of the invention. In FIG. 7, atstep 700, a data source, such as a log file, is opened for reading. Atstep 705, the program checks to see if there are any hits left in thelog file for processing. If there are, the next hit is read from the logfile at step 710. At step 715, the current hit is assigned to a visitorsession, described with reference to FIG. 8.

Turning to FIG. 8, in step 500, the hit is analyzed to determine theaddress of the visitor who generated the hit. This information is easilydetermined, as the structure of the hit is well documented, as describedabove and depicted in FIGS. 2-4.

Still looking at FIG. 8, once the current hit's visitor address has beendetermined, at step 505 the program scans the visitor session database.A visitor session is a sequence of hits received from a single visitor.The sequence extends between the first hit until a predetermined timeafter the most recent hit has lapsed without further hits. Although theterm visitor was used to describe the qualification profiles and adcampaigns with reference to FIGS. 5 and 6, the program in fact tracksvisitor sessions. Each visitor session, of course , is associated with asingle visitor. All visitor sessions are maintained in a database. Atstep 510 the program checks to see if the visitor address exists in thevisitor session database. If the visitor address is new to the visitorsession database, then at step 515 a new entry to the visitor sessiondatabase is created, using the information from the hit (which includeswhat, if any, advertising campaign the visitor followed, discussedbelow). If the visitor address already existed at step 510, then at step520 the date and time of the current hit are determined. At step 525 theprogram checks to see how long it has been since the visitor's lastactivity, i.e., since the last hit was received from that visitor. If ithas been longer than the predetermined session time, then the sessionfor that visitor is concluded. When the same visitor revisits the siteafter their session was concluded, then step 515 is performed and a newvisitor record in the visitor database is created. But if the visitorsession is still active, i.e., the predetermined time from the last hithas not lapsed when the next hit arrives, step 530 is performed and thevisitor database is updated according to the current hit. This includesupdating the visitor's last date and time of activity and updating thelist of actions the visitor has performed.

After completion of the routine of FIG. 8, control then returns to step705 in FIG. 7 to process additional hits as described above. If,however, there are no hits left in the log file for processing at step705, control moves to step 720, where the program checks to see if thereare any visitor sessions that need to be assigned to qualificationprofiles. If there are visitor sessions left to process, then step 725reads the history for one of the remaining visitor sessions. Step 730,in a routine depicted in FIG. 9, then assigns the current visitorsession to all qualification profiles for which it qualifies.

Referring to FIG. 9, at step 600, the complete history of the visitorsession to which the current hit was assigned is read.

There are currently two ways that the program can identify whichparticular advertising campaign a visitor followed to the site. One isby referrer. A referrer is the web page that held the link the visitorfollowed to reach the web site. For example, referring back to FIGS.1-4, the referrer is the web page http://www.portal.com/somepage.html,which referred the remote visitor 12 to the first web site 10,www.example.com.

The second way the program can identify which particular advertisingcampaign a visitor followed to the site is by the entry page. The entrypage is the first web page on server 10 visited; in the case ofadvertising campaigns, each advertising campaign can have a differententry page (all of which differ from the entry page used by visitors whoare not following an advertising campaign). For example, referring toFIG. 3, the entry page on server 10 is the web pagehttp://www.example.com/portal_ad.htm.

Regardless of the method the advertising campaign uses to determine whatadvertisement a visitor followed, if any, that information is availablefrom the first hit a visitor generates at the web site. Referring backto FIG. 3, entry 11 a shows both the requested page (“GET/portal_ad.htmHTTP/1.0”) and the referring site(“http://www.portal.com/somepage.htm”). Thus, the program uses the firsthit a visitor generates to determine what advertising campaign theyfollowed. The entry page and referrer is assigned in step 601, and—based on this data—the ad campaign is assigned in step 603.

At step 605 the first qualification profile is read. Again, there is noconcern of trying to read a non-existent object: if no qualificationprofiles exist, the program will not be used, since no analysis isneeded. At step 610 the visitor session history (read at step 600) iscompared with the requirements of the qualification levels of thecurrent qualification profile (read at step 605). At step 615, theprogram checks to see if the visitor session has met all therequirements for any of the qualification levels of the currentqualification profile. If it has, then at step 620 the visitor sessionis added to the appropriate qualification levels of the currentqualification profile. Note that, in the preferred embodiment, withineach qualification profile, the qualification levels are ranked, and avisitor session is added to the highest-ranking qualification level ofeach qualification profile for which the visitor session meets therequirements, but this limitation is not always necessary. (Thepreferred embodiment also ranks qualification levels: the visitor isadded to the highest ranked qualification level—within eachqualification profile—for which the visitor qualifies.) At step 625,whether or not the visitor session met the requirements for any of thequalification levels of the current qualification profile, the programchecks to see if there are any more qualification profiles defined. Ifthere are, then at step 630 the next qualification profile is read andcontrol returns to step 610. Otherwise, the processing of the currenthit is finished, and control reverts to step 720 in FIG. 7.

If there are remaining visitor sessions to process, the FIG. 7 routinedoes so in steps 725 and 730 as described above. Conversely, if thereare no remaining visitor sessions left to process at step 720, controlis transferred to step 735, where the return-on-investment (ROI)analysis is performed.

Before describing the ROI analysis consideration will first be given tothe raw data and its format that exists after the routines of FIGS. 7-9are run as described above. For each ad campaign, the number of visitorsessions that met each level of each qualification profile is stored intables, as depicted in the Table A example:

TABLE A QP1 QP2 L1 L2 L3 L1 L2 NQ ADC1 4 6 7 3 5 35 ADC2 5 9 12 6 11 63ADC3 1 0 2 3 4 20 NADC 21 35 46 28 39 211 QPn—Qualification Profile nADCn—Ad Campaign n NQ—Not qualified for any QP NADC—No Ad CampaignLn—Qualification Level n

Those visitor sessions that do not result from an as campaign that isbeing monitored are classified in the No-Ad-Campaign category. Thosevisitor sessions that do not meet the criteria of any of theQualification Profiles are categorized in the NQ column. For eachreferrer, the number of visitor sessions that met each qualificationprofile is stored in tables, as depicted in the Table B example:

TABLE B QP1 QP2 L1 L2 L3 L1 L2 NQ R1 2 7 4 2 6 25 R2 4 2 16 2 11 83 R3 23 7 0 0 10 NR 31 32 43 22 31 124 Rn—Referrer n NR—No Referrer

Those visitor sessions that have no referrer are classified in theNo-Referrer category. One instance in which a visitor session has noreferrer arises when a visitor types in the monitored web-site address(typically a home page URL) and goes directly to the web site. Allvisitor sessions that do have a referrer are associated with thereferrer. In other words, all referrers are tracked. When a new referreris encountered, a new entry in the table is made.

The ROI analysis uses the following raw data: Tables A and B; the periodof the report, which is the time of the first to the last recordsprocessed; the qualification profiles (FIG. 5); the ad campaigns (FIG.6); and the nonqualified visitor value assigned to the visitor sessionsthat wind up in the NQ category.

Tables 1-9 below show sample output tables that can be generated by ROIanalysis. These tables cross-correlate information from the raw data invarious ways. For example, Table 1 shows the number of visitors referredto the web server from six different referring URLs, along with thevalue of the visitors referred. For each referrers, the number ofvisitor sessions is shown, as well as the total visitor sessions for allreferrers. For each referrers, the value of the visitor sessions foreach qualification level is calculated by multiplying the value fromthat level (FIG. 5) times the visitor sessions meeting the criteria forthe qualification profile. Total value of the visitor sessions for allreferrers is the sum of these products.

Various additional variables can be calculated using the raw data, anddisplayed in tables, additional tables, like Tables 2-9 below. Althougheach calculation is not described, the manner of using the raw data togenerate each of the calculated variables in the tables can be easilyinferred from the name of the variable in view of the following briefdescription of the table.

Table 2 shows the number of visitors in each qualification profile (inthis instance downloading of specified products Product 1, Product 2,and Product 3), arranged by qualification level and referring URL. Table3 shows the number of visitors from each of six advertising campaigns,broken down into qualified and non-qualified visitors, and the valuesthe visitors generate. Although not shown, it should be noted that alluntracked visitors, i.e., those not associated with a tracked adcampaign, can be summarized in the same categories as those shown inTable 3 for tracked visitors. Table 4 shows the number of visitorsreferred from each advertising campaign, sorted by qualification profileand qualification level. Table 5 shows visitors assigned to eachqualification profile and the value they generate sorted by advertisingcampaign. Table 6 shows visitors for each qualification profile and thevalue they generated, sorted by qualification level and advertisingcampaign. Table 7 shows the cost of each of the six advertisingcampaigns and the cost per visitor for each visitor referred from therespective advertising campaigns. Table 8 shows the return on investmentfrom each advertising campaign, where the return on investment is basedon the value of the visitors referred from the respective advertisingcampaigns less the cost of each campaign. Finally, Table 9 shows thedaily return on investment shown in Table 8.

TABLE 1 Summary of Results by Referrer Value of Visitor VisitorsReferrer Sessions Referred 1 http://www.yahoo.com/ 1303 $3,229.50 2http://www.excite.com/ 980 $2,238.00 3 http://www.builder.com/ 900$2,511.50 4 http://www.intergreat.com/ 605 $1,540.50 5http://www.lycos.com/ 443 $1,194.00 6 http://www.hotwired.com/ 257  $652.50 Total 4488 $11,366.00 

TABLE 2 Top Referring Sites By Qualification Profile and QualificationLevel Qualification Qualification Visitor % of Profile Level ReferrerSessions Total 1 Product 1 Fully Qualified http://www.yahoo.com/ 2434.29% http://www.builder.com/ 21 30.00% http://www.excite.com/ 9 12.86%http://www.intergreat.com/ 8 11.43% http://www.lycos.com/ 6 8.57%http://www.hotwired.com/ 2 2.86% Subtotal 70 100.00% Moderatelyhttp://www.builder.com/ 53 34.64% Qualified http://www.yahoo.com/ 4126.80% http://www.excite.com/ 20 13.07% http://www.lycos.com/ 18 11.76%http://www.intergreat.com/ 12 7.84% http://www.hotwired.com/ 9 5.88%Subtotal 153 100.00% Minimally http://www.builder.com/ 72 30.90%Qualified http://www.yahoo.com/ 60 25.75% http://www.intergreat.com/ 3213.73% http://www.excite.com/ 30 12.88% http://www.lycos.com/ 22 9.44%http://www.hotwired.com/ 17 7.30% Subtotal 233 100.00% Group Subtotal456 100.00% 2 Product 2 Fully Qualified http://www.yahoo.com/ 13 34.21%http://www.builder.com/ 8 21.05% http://www.intergreat.com/ 6 15.79%http://www.excite.com/ 5 13.16% http://www.lycos.com/ 4 10.53%http://www.hotwired.com/ 2 5.26% Subtotal 38 100.00% Moderatelyhttp://www.lycos.com/ 23 31.51% Qualified http://www.builder.com/ 1520.55% http://www.intergreat.com/ 14 19.18% http://www.yahoo.com/ 1216.44% http://www.excite.com/ 9 12.33% http://www.hotwired.com/ 0 0.00%Subtotal 73 100.00% Minimally http://www.yahoo.com/ 32 28.57% Qualifiedhttp://www.builder.com/ 23 20.54% http://www.intergreat.com/ 26 23.21%http://www.lycos.com/ 15 13.39% http://www.excite.com/ 10 8.93%http://www.hotwired.com/ 6 5.36% Subtotal 112 100.00% Group Subtotal 223100.00% 3 Product 3 Fully Qualified http://www.builder.com/ 12 33.33%http://www.lycos.com/ 9 25.00% http://www.intergreat.com/ 7 19.44%http://www.yahoo.com/ 4 11.11% http://www.hotwired.com/ 13 8.33%http://www.excite.com/ 1 2.78% Subtotal 36 100.00% Moderatelyhttp://www.intergreat.com/ 19 29.69% Qualified http://www.builder.com/15 23.44% http://www.lycos.com/ 13 20.31% http://www.hotwired.com/ 1117.19% http://www.yahoo.com/ 6 9.38% http://www.excite.com/ 0 0.00%Subtotal 64 100.00% Minimally http://www.builder.com/ 28 26.42%Qualified http://www.intergreat.com/ 22 20.75% http://www.lycos.com/ 2119.81% http://www.hotwired.com/ 16 15.09% http://www.yahoo.com/ 1312.26% http://www.excite.com/ 6 5.66% Subtotal 106 100.00% GroupSubtotal 206 100.00% Total 885 100.00%

TABLE 3 Summary of Results by Ad Campaign Non- Non- Qualified QualifiedQualified Qualified Visitor Qualified Total Visitor Visitor VisitorVisitor Average Visitors Visitor Average Total Visitor Ad CampaignSessions Value Sessions Value Total Value Sessions Value Value 1 Yahoo!Banner 1098 $2,196.00 205 $5.04 $1,033.50 1303 $2.48 $3,229.50 2Builder.com Banner 653 $1,306.00 247 $4.88 $1,205.50 900 $2.79 $2,511.503 Excite Banner 890 $1,780.00 90 $5.09   $458.00 980 $2.28 $2,238.00 4Intergreat Banner 459   $918.00 146 $4.26   $622.50 605 $2.55 $1,540.505 Lycos Banner 312   $624.00 131 $4.35   $570.00 443 $2.70 $1,194.00 6HotWired WebMonkey 193   $386.00 64 $4.16   $266.50 257 $2.54   $652.50Banner Total 3605 $7,210.00 883 $4.71 $4,156.00 4488 $2.53 $11,366.00 

TABLE 4 Results by Ad Campaign, Qualification Profile and QualificationLevel % of Qualification Qualification Visitor Visitor Ad CampaignProfile Level Sessions Sessions Value 1 Yahoo! Banner Product 1 FullyQualified 24 1.84% $192.00  Moderately Qualified 41 3.15% $246.00 Minimally Qualified 60 4.60% $300.00  Product 2 Fully Qualified 13 1.00%$78.00 Moderately Qualified 12 0.92% $48.00 Minimally Qualified 32 2.46%$96.00 Product 3 Fully Qualified 4 0.31% $20.00 Moderately Qualified 60.46% $21.00 Minimally Qualified 13 1.00% $32.50 Non-Qualified 109884.27% $2,196.00   Subtotal 1303 100.00% $3,229.50   2 Excite BannerProduct 1 Fully Qualified 9 0.92% $72.00 Moderately Qualified 20 2.04%$120.00  Minimally Qualified 30 3.06% $150.00  Product 2 Fully Qualified5 0.51% $30.00 Moderately Qualified 9 0.92% $36.00 Minimally Qualified10 1.02% $30.00 Product 3 Fully Qualified 1 0.10%  $5.00 ModeratelyQualified 0 0.00%  $0.00 Minimally Qualified 6 0.61% $15.00Non-Qualified 890 90.82% $1,780.00   Subtotal 980 100.00% $2,238.00   3Builder.com Banner Product 1 Fully Qualified 21 2.33% $168.00 Moderately Qualified 53 5.89% $318.00  Minimally Qualified 72 8.00%$360.00  Product 2 Fully Qualified 8 0.89% $48.00 Moderately Qualified15 1.67% $60.00 Minimally Qualified 23 2.56% $69.00 Product 3 FullyQualified 12 1.33% $60.00 Moderately Qualified 15 1.67% $52.50 MinimallyQualified 28 3.11% $70.00 Non-Qualified 653 72.56% $1,306.00   Subtotal900 100.00% $2,511.50   4 Intergreat Banner Product 1 Fully Qualified 81.32% $64.00 Moderately Qualified 12 1.98% $72.00 Minimally Qualified 325.29% $160.00  Product 2 Fully Qualified 6 0.99% $36.00 ModeratelyQualified 14 2.31% $56.00 Minimally Qualified 26 4.30% $78.00 Product 3Fully Qualified 7 1.16% $35.00 Moderately Qualified 19 3.14% $66.50Minimally Qualified 22 3.64% $55.00 Non-Qualified 459 75.87% $918.00 Subtotal 605 100.00% $1,540.50   5 Lycos Banner Product 1 FullyQualified 6 1.35% $48.00 Moderately Qualified 18 4.06% $108.00 Minimally Qualified 22 4.97% $110.00  Product 2 Fully Qualified 4 0.90%$24.00 Moderately Qualified 23 5.19% $92.00 Minimally Qualified 15 3.39%$45.00 Product 3 Fully Qualified 9 2.03% $45.00 Moderately Qualified 132.93% $45.50 Minimally Qualified 21 4.74% $52.50 Non-Qualified 31270.43% $624.00  Subtotal 443 100.00% $1,194.00   6 HotWired WebMonkeyProduct 1 Fully Qualified 2 0.78% $16.00 Banner Moderately Qualified 93.50% $54.00 Minimally Qualified 17 6.61% $85.00 Product 2 FullyQualified 0 0.00%  $0.00 Moderately Qualified 0 0.00%  $0.00 MinimallyQualified 6 2.33% $18.00 Product 3 Fully Qualified 3 1.17% $15.00Moderately Qualified 11 4.28% $38.50 Minimally Qualified 16 6.23% $40.00Non-Qualified 193 75.10% $386.00  Subtotal 257 100.00% $652.50  Total4488 100.00% $11,366.00   

TABLE 5 Results by Qualification Profile Qualified % of QualificationVisitor Visitor % of Profile Ad Campaign Sessions Sessions Value Value 1Product 1 Builder.com Banner 146 32.02% $846.00 32.01% Yahoo! Banner 12527.41% $738.00 27.92% Excite Banner 59 12.94% $342.00 12.94% IntergreatBanner 52 11.40% $296.00 11.20% Lycos Banner 46 10.09% $266.00 10.06%HotWired WebMonkey Banner 28 6.14% $155.00  5.86% Subtotal 456 100.00%$2,643.00   100.00%  2 Product 2 Yahoo! Banner 57 25.79% $222.00 26.30%Builder.com Banner 46 20.81% $177.00 20.97% Intergreat Banner 46 20.81%$170.00 20.14% Lycos Banner 42 19.00% $161.00 19.08% Excite Banner 2410.86%  $96.00 11.37% HotWired WebMonkey Banner 6 2.71%  $18.00  2.13%Subtotal 221 100.00% $844.00 100.00%  3 Product 3 Builder.com Banner 5526.70% $182.50 27.28% Intergreat Banner 48 23.30% $156.50 23.39% LycosBanner 43 20.87% $143.00 21.38% HotWired WebMonkey Banner 30 14.56% $93.50 13.98% Yahoo! Banner 23 11.17%  $73.50 10.99% Excite Banner 73.40%  $20.00  2.99% Subtotal 206 100.00% $669.00 100.00% 

TABLE 6 Results by Qualification Profile and Qualification Level % of %of Qualification Qualification Visitor Visitor Visitor Visitor ProfileLevel Ad Campaign Sessions Sessions Value Value 1 Product 1 FullyQualified Yahoo! Banner 24 34.29% $192.00 34.29% Builder.com Banner 2130.00% $168.00 30.00% Excite Banner 9 12.86%  $72.00 12.86% IntergreatBanner 8 11.43%  $64.00 11.43% Lycos Banner 6 8.57%  $48.00 8.57%HotWired WebMonkey Banner 2 2.86%  $16.00 2.86% Subtotal 70 100.00%$560.00 100.00% Moderately Builder.com Banner 53 34.64% $318.00 34.64%Qualified Yahoo! Banner 41 26.80% $246.00 26.80% Excite Banner 20 13.07%$120.00 13.07% Lycos Banner 18 11.76% $108.00 11.76% Intergreat Banner12 7.84%  $72.00 7.84% HotWired WebMonkey Banner 9 5.88%  $54.00 5.88%Subtotal 153 100.00% $918.00 100.00% Minimally Builder.com Banner 7230.90% $360.00 30.90% Qualified Yahoo! Banner 60 25.75% $300.00 25.75%Intergreat Banner 32 13.73% $160.00 13.73% Excite Banner 30 12.88%$150.00 12.88% Lycos Banner 22 9.44% $110.00 9.44% HotWired WebMonkeyBanner 17 7.30%  $85.00 7.30% Subtotal 233 100.00% $1,165.00   100.00%Total 456 — $2,643.00   — 2 Product 2 Fully Qualified Yahoo! Banner 1336.11%  $78.00 36.11% Builder.com Banner 8 22.22%  $48.00 22.22%Intergreat Banner 6 16.67%  $36.00 16.67% Excite Banner 5 13.89%  $30.0013.89% Lycos Banner 4 11.11%  $24.00 11.11% HotWired WebMonkey Banner 00.00%  $0.00 0.00% Subtotal 36 100.00% $216.00 100.00% Moderately LycosBanner 23 31.51%  $92.00 31.51% Qualified Builder.com Banner 15 20.55% $60.00 20.55% Intergreat Banner 14 19.18%  $56.00 19.18% Yahoo! Banner12 16.44%  $48.00 16.44% Excite Banner 9 12.33%  $36.00 12.33% HotWiredWebMonkey Banner 0 0.00%  $0.00 0.00% Subtotal 73 100.00% $292.00100.00% Minimally Yahoo! Banner 32 28.57%  $96.00 28.57% QualifiedIntergreat Banner 26 23.21%  $78.00 23.21% Builder.com Banner 23 20.54% $69.00 20.54% Excite Banner 10 8.93%  $30.00 8.93% Lycos Banner 1513.39%  $45.00 13.39% HotWired WebMonkey Banner 6 5.36%  $18.00 5.36%Subtotal 112 100.00% $336.00 100.00% Total 221 — $844.00 — 3 Product 3Fully Qualified Builder.com Banner 12 33.33%  $60.00 33.33% Lycos Banner9 25.00%  $45.00 25.00% Intergreat Banner 7 19.44%  $35.00 19.44% Yahoo!Banner 4 11.11%  $20.00 11.11% HotWired WebMonkey Banner 3 8.33%  $15.008.33% Excite Banner 1 2.78%  $5.00 2.78% Subtotal 36 100.00% $180.00100.00% Moderately Intergreat Banner 19 29.69%  $66.50 29.69% QualifiedBuilder.com Banner 15 23.44%  $52.50 23.44% Lycos Banner 13 20.31% $45.50 20.31% HotWired WebMonkeyBanner 11 17.19%  $38.50 17.19% Yahoo!Banner 6 9.38%  $21.00 9.38% Excite Banner 0 0.00%  $0.00 0.00% Subtotal64 100.00% $224.00 100.00% Minimally Builder.com Banner 28 26.42% $70.00 26.42% Qualified Intergreat Banner 22 20.75%  $55.00 20.75%Lycos Banner 21 19.81%  $52.50 19.81% HotWired WebMonkey Banner 1615.09%  $40.00 15.09% Yahoo! Banner 13 12.26%  $32.50 12.26% ExciteBanner 6 5.66%  $15.00 5.66% Subtotal 106 100.00% $265.00 100.00% Total206 — $669.00 —

TABLE 7 Ad Campaign Costs Total Qualified Cost Per Cost This VisitorCost Per Visitor Qualified Ad Campaign Period Sessions Visitor SessionsVisitor 1 Yahoo! Banner $1,250.00 1303 $0.96 205 $6.10 2 Excite Banner  $433.00 980 $0.44 90 $4.81 3 Builder.com Banner $1,250.00 900 $1.39247 $5.06 4 Intergreat Banner   $200.00 605 $0.33 146 $1.37 5 LycosBanner $2,000.00 443 $4.51 131 $15.27  6 HotWired WebMonkey Banner$1,250.00 257 $4.86 64 $19.53  Total $6,383.00 4488 $1.42 883 $7.23

TABLE 8 Ad Campaign Return on Investment # Days Cost of Total Visitor %ROI Active This Campaign Value ROI This This Ad Campaign Period ThisPeriod Generated Period Period 1 Yahoo! Banner 7 $1,250.00 $3,229.50$1,979.50 158.36% 2 Excite Banner 16    $433.00 $2,238.00 $1,805.00416.86% 3 Intergreat Banner 5   $200.00 $1,540.50 $1,340.50 670.25% 4Builder.com Banner 7 $1,250.00 $2,511.50 $1,261.50 100.92% 5 HotWiredWebMonkey Banner 9 $1,250.00   $652.50  ($597.50) −47.80% 6 Lycos Banner12  $2,000.00 $1,194.00  ($806.00) −40.30% Total — $6,383.00 $11,366.00 $4,983.00 78.07%

TABLE 9 Ad Campaign Daily Return on Investment # Days Cost VisitorActive This Per Value Per ROI Per Ad Campaign Period Day Day Day 1Yahoo! Banner 7 $178.57 $461.36 $282.79 2 Intergreat Banner 5  $40.00$308.10 $268.10 3 Builder.com 7 $178.57 $358.79 $180.21 Banner 4 ExciteBanner 16   $27.06 $139.88 $112.81 5 HotWired 9 $138.89  $72.50 ($66.39) WebMonkey Banner 6 Lycos Banner 12  $166.67  $99.50  ($67.17)Total — $563.09 $1,340.62   $777.52

Turning now to FIGS. 10 and 11, illustrated therein is a flow chart,indicated generally as program 800, depicting steps in a computerprogram constructed in accordance with the present invention. As is thecase with the previously described programs, program 800 operates onserver 10 in the present embodiment of the invention, although this isnot required.

Program 800 is advantageous in that it tracks only criteria of interestfor each active visitor session and—at the conclusion of eachsession—determines whether the visitor whose session was monitored meetsthe criteria for each qualification profile. And if so, the programassigns the visitor accordingly. This naturally requires less processingand memory resources than program 700.

Turning now to FIG. 10, in step 801, the program checks to see if thereare more hits in the log file. If so, in step 805, the program reads thenext hit generated by interaction of a remote visitor with server 10. Ifit is a hit from a new visitor, i.e., there is no visitor session activefor the particular visitor who produced the current hit, a new visitorsession is created in step 815. At the same time, the visitor session isassigned to an entry page and referrer and to an ad campaign in steps817, 819, respectively, as described above. Thereafter, a checklist isassigned to the new visitor session in 820. The checklist is merely alist of all of the requirements for each qualification level in eachqualification profile. In the case where criteria for each of therequirements are selected web pages identified by URLs on the site, thechecklist assigned to each visitor session in step 820 is merely a listof all of those URLs.

Returning again to step 810, if there is a visitor session active forthe particular visitor who produced the current hit, that visitor'ssession is updated. When a visitor session is updated responsive to acurrent hit from that visitor, the information in the hit is examined todetermine whether or not any of the URLs on the checklist were visitedand, if so, that URL is flagged. In the present embodiment of theinvention, the same checklist is assigned to each visitor session.

This process begins in step 824 when the hit is examined to determinewhether or not the page is on the checklist. If so, that page on thelist is flagged in step 825, and a session timing clock, which is uniqueto each visitor session, is reset and started in step 830. If not, theprogram moves from step 824 to step 830 to reset and start the clockassociated with that visitor session. In other words, every hit from thevisitor resets and starts the clock in step 830 whether the visitorviews a page on the checklist or not. After resetting and starting theclock, the program—in step 835—checks each open visitor session. Foreach session, in step 840, the program determines whether the clockassociated with that visitor session has timed out. Put differently, itdetermines whether the predetermined time used to define each visitorsession has lapsed without another hit from that visitor. If not, theprogram leaves the visitor session open and returns control to step 835to check the next visitor session. If the session has lapsed, theprogram closes that session in accordance with FIG. 11, which isdescribed shortly, and returns control to step 835 to check the nextsession. After all current sessions are checked using the routine insteps 835, 840, and FIG. 11, control is transferred again to step 801 tocheck for more hits, as described above.

When there are no more hits in the log file, control transfers todecision box 837, which checks all remaining open visitor sessions andcloses each in accordance with FIG. 11. Each time a session is closed,control again returns to step 837 to close the next visitor sessionuntil all are closed by the FIG. 11 routine. Control then goes to step839, wherein the ROI analysis is performed in the same manner asdescribed in connection with the embodiment of FIGS. 7-9.

Turning now to FIG. 11, consideration will be given to the manner inwhich a visitor session is closed. With reference to FIG. 10, thisoccurs after step 840, when each session expires, and after step 801,when there are no more hits to process and all remaining visitorsessions are consequently closed. In steps 845, 850, the session isclosed, and the checklist associated with the session is examined todetermine which pages the visitor viewed during the visitor session. Instep 855 the flagged pages are compared with the pages designated ineach qualification level within each qualification profile. In step 860,the visitor is assigned to the highest qualification level, likequalification level 315 in FIG. 5, in each qualification profile, likequalification profile 305 in FIG. 5, for which she qualifies. This isaccomplished in the same manner as described in connection with FIG. 9.Next, in step 865, Tables A and B are updated accordingly. It is fromthese tables that data are extracted, as described above, to producetables like those shown in Tables 1-9 above.

Having illustrated and described the principles of our invention in apreferred embodiment thereof, it should be readily apparent to thoseskilled in the art that the invention can be modified in arrangement anddetail without departing from such principles. We claim allmodifications coming within the spirit and scope of the accompanyingclaims.

1. A method for associating visitor profiles with products or servicescomprising: establishing a visitor profile for each visitor to one ormore web servers; defining a plurality of qualification levels tovisitors of one or more URLs associated with the one or more webservers; associating each of the plurality of qualification levels withproducts or services, at least two of the qualification levels beingassociated with different levels of interest in the same product orservice; and analyzing visitor profiles to associate each visitor withone or more of the plurality of qualification levels, wherein theestablishing, defining, associate, and analyzing steps take place withina computer.
 2. The method of claim 1 wherein each qualification levelrepresents a different level of interest in a product or service offeredfor sale at the one or more web servers.
 3. The method of claim 1wherein analyzing visitor profiles comprises: determining which URLs onthe one or more web servers each visitor visited; and assigning eachvisitor to one or more of the plurality of qualification levels based onthe URLs visited by the visitor.
 4. The method of claim 1 whereinassociating each visitor with one or more of the plurality ofqualification level comprises: comparing each visitor with criteriadefining each qualification level; and associating each visitor with oneor more qualification levels whose criteria the visitor meets.
 5. Themethod of claim 1 wherein establishing a visitor profile for eachvisitor comprises creating a visitor session to record hits associatedwith the visitor, where each hit corresponds to data exchanged betweenthe one or more web servers and a computing device associated with thevisitor.
 6. The method of claim 1 wherein said method further comprisesassigning a quantitative value to each of the plurality of qualificationlevels.
 7. The method of claim 1 wherein analyzing visitor profilescomprises: determining which URLs on the one or more web servers eachvisitor visited; and assigning each visitor to one or more of theplurality of qualification levels based on the URLs visited by thevisitor.
 8. The method of claim 1 wherein associating each visitor withone or more of the plurality of qualification level comprises: comparingeach visitor with criteria defining each qualification level; andassociating each visitor with one or more qualification levels whosecriteria the visitor meets.
 9. The method of claim 1 whereinestablishing a visitor profile for each visitor comprises creating avisitor session to record hits associated with the visitor, where eachhit corresponds to data exchanged between the one or more web serversand a computing device associated with the visitor.
 10. The method ofclaim 1 wherein said method further comprises assigning a quantitativevalue to each of the plurality of qualification levels.